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(54) System and method for determining representative frames of video captured by a video 
camera 



(57) A system and method of determining repre- 
sentative frames (56) of a video sequence to form a vis- 
ual table of contents for a video tape (34) is disclosed. A 
representative frame is identified on the occurence of 
an event, such as the beginning of a recording 
sequence, which may follow a record command, a 
pause-release command, some form of special effect 
command, or a user-initiated marking command. The 
representative frame, and location data associated 
therewith, may be marked along the length of a video 
tape (34) in a camcorder (12) or VCR, or, the represent- 
ative frame (56) may be identified and stored, along with 
the location data therefor, on a MIC (18). In the case 
where the representative frame location data is stored 
on the tape (34). a scanning mechanism (22) is pro- 
vided for scanning the recorded video tape to locate the 
representative frames (56). In either form of the inven- 
tion, multiple representative frames and their location 
data are placed in a storage mechanism (24), either 
temporarily or permanently, and then recorded at a sin- 
gle location, either at the beginning of the video tape 
(34) or in the MIC (18). A peripheral storage mechanism 
(28) may also be provided, in the form of a computer, 
which may be used to add additional information to the 
representative frame (56), and which may also be used 
in editing the video tape (34). 
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Description 

Background of the Invention 

This invention relates to manipulation of video and s 
data on a video tape, and specifically to the identifica- 
tion and the collecting of representative frames of video 
to provide a visual table of contents for a video tape. 

Standards of consumer-use digital video cassette 
recorder (DVCR) provide specifications for the record- io 
ing of video and audio signals on a digital video tape, as 
well as providing for the recording of peripheral informa- 
tion on the tape or, in some instances, for the recording 
of information on what is referred to as memory-in-cas- 
sette (MIC). This is a significant advance over the lim- 15 
ited information which may be placed on an analog 
tape. As will be explained later herein, consumer-use 
DVCR standards provide for audio auxiliary data, video 
auxiliary data, a variety of mandatory and optional data, 
which is recorded in the form of data packets (PACKs), 20 
which are included in groups broadly identified as sub- 
code data and system data. 

Brief Summary of the Invention 

The instant invention defines a system and method 
of determining a representative frame of a video 
sequence. This representative frame is identified on the 
occurrence of an event, such as the beginning of a 
recording sequence, which may follow a record com- 
mand, a pause-release command, some form of special 
effect command, or an event-specific signal initiated by 
the user. The representative frame, and location data 
associated therewith, may be marked along the length 
of a video tape in a VCR or a camcorder, or, the repre- 
sentative frame may be identified and stored, along with 
the location data therefor, on a MIC, which is a memory 
device suitable for storing a large quantity of data, such 
as a flash memory device. In the case where the repre- 
sentative frame location data is initially marked and then 
stored on the tape, a scanning mechanism is provided 
for scanning the recorded video tape to locate the rep- 
resentative frames. In either form of the invention, multi- 
ple representative frames and their location data are 
placed in a storage mechanism, either temporarily or 
permanently, and then recorded at a single location, 
either at the begining of the video tape or in the MIC. A 
peripheral storage mechanism may also be provided, in 
the form of a computer, which may be used to insert 
additional information in the representative frame, and 
which may also be used in editing and archiving the 
video tape contents. A mechanism for marking repre- 
sentative frames is located in a camcorder, or in a video 
cassette recorder. 

It is an object of the invention to provide a system 55 
and method for constructing a visual table of contents 
for a video tape. 

Another object of the invention is to provide a visual 
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table of contents which is stored on the video tape. 

Another object of the invention is to provide a visual 
table of contents which is stored on a memory device in 
the cassette for a video tape. 

These and other objects and advantages of the 
invention will become more fully apparent as the 
description which follows is read in conjunction with the 
drawings. 

Brief Description of the Several Views of the Drawing s 

Fig. 1 is a block diagram of the system of the inven- 
tion. 

Fig. 2 is a schematic representation of a digital 
video tape. 

Fig. 3 is a block diagram of the components of the 
system and method. 

Fig. 4 is a monitor having an index frame memory 
display depicted thereon. 

Fig. 5 is a depiction of the MIC storage mechanism. 

Detailed Description of the Preferred Embodiments 

Referring now to the drawings, initially to Fig. 1, the 
system for determining representative frames of video 
captured by a video camera is depicted in block-dia- 
gram form generally at 10. The representative frames of 
video are collected and assembled to provide a visual 
table of contents for the video tape. System 10 includes 
a camcorder 12, having a recording mechanism 13 and 
a marking module 14 located therein. Marking module 
14, in this embodiment of the invention, is operational to 
determine a representative frame of video during cap- 
ture of a video image by camcorder 12. Camcorder 12 
is constructed and arranged to receive a video tape cas- 
sette 16, which may, in one embodiment of the inven- 
tion, include a memory-in-cassette (MIC) mechanism 
18. 

System 10, in this embodiment, includes a docking 
station 20, which receives camcorder 12 for further 
processing of the video tape. Docking station 20 
includes a scanning mechanism 22, a resizing and 
arrangement mechanism 23 and a storage mechanism 
24. A peripheral storage mechanism 26 may be pro- 
vided in the form of a personal computer, and may have 
an additional storage mechanism 28 and a special 
effects (F/X) mechanism 30 located therein, which pro- 
vides an editing and archiving function. Other embodi- 
ments of the system may include a camcorder which 
includes the components of docking station 20.. 

Referring now to Fig. 2, a length of video tape is 
depicted generally at 34. The structure of the tape 
according to the DVCR standard is as follows: each 
frame contains video, audio and other data, which is 
stored in multiple tracks. Tape 34 moves over a record- 
ing head in the direction indicated by arrow 36, while the 
recording head rotates in the direction indicated by 
arrow 38. It will be understood by those of skill in the art 
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that tape 34 is being viewed from the magnetic media 
side thereof The data for each frame is stored in multiple 
A/V tracks, such as first A/V track 40 and second A/V 
track 41. The DVCR standard applicable in countries 
using a 60Hz power supply requires 525 lines of resolu- 5 
tion, located in 10 A/V tracks per frame. The DVCR 
standard in 50Hz countries requires 625 lines of resolu- 
tion, located in 12 A/V tracks per frame. The DVCR 
standard also provides for a first optional track 42 and a 
second optional track 43. w 

Each A/V track has four main sectors: (i) an insert 
and track information (ITI) sector 44, (ii) an audio sector 
46, having an audio auxiliary (AAUX) sector 47 therein, 
(iii) a video sector 48, having a video auxiliary (VAUX) 
sector 49 therein, and (iv) a sub-code sector 50. Com- 15 
pressed audio and video data are recorded on tape 34 
in audio sector 46 and video sector 48, respectively 
Peripheral, or optional, information (data) may be 
recorded in AAUX sector 47, VAUX sector 49 and sub- 
code sector 50. Each of the AAUX, VAUX and sub-code 20 
sectors, and MIC 18, is further divided into "common 
optional" and "maker's optional" areas. These optional 
areas, particularly the maker's optional areas, may be 
used as a storage location for signals, or reference 
marks, which are placed on video tape 34, or MIC 1 8, to 25 
mark representative frames of video. 

The DVCR standard provides for what are referred 
to as maker-option PACKS (MOPs). A PACK is a unit of 
some data groups. A PACK is five bytes long and 
includes a one-byte header PACK (PC 0 ) having upper 30 
and lower four-bit blocks, and a four-byte data PACK 
(PCi - PC 4 ). MOPs are located in the data section of a 
synchronization block, which is located in sub-code sec- 
tor 50 of ah A/V track. Each sub-code sector has twelve 
sub-code sync blocks, each having a 12 byte length, five 35 
bytes of which are sub-code data. The sub-code area is 
divided into MAIN and OPTIONAL areas, with part of 
the optional areas, identified as the maker's optional 
area, being available to manufacturers for use as they 
see fit. 40 

Referring to Fig. 3, the method of the invention is 
depicted generally at 60. The first A/V track of a repre- 
sentative frame, such as track 40, is identified and 
marked 62 on the tape upon the occurrence of a prede- 
termined event. The location of the first A/V track, T fl of 45 
a representative frame is identified by arrow 56 in Fig. 2, 
which arrow is symbolic of a reference mark 64 which is 
placed in the fist A/V track of a representative frame. As 
previously noted, reference mark 64 may be placed in 
the AAUX sector, the VAUX sector, the sub-code, or, if so 
so equipped, the track number of reference mark 64 
location may be placed in MIC 18 of the tape cassette. 
The preferred location of reference mark 64 is in the 
maker's optional area of the sub-code sector, as this 
location provides the most efficient retrieval of data 55 
when compared to the AAUX and VAUX sectors. 

The predetermined event which initiates the identi- 
fication of a representative frame may be the beginning 



of a recording sequence, which is initiated by pressing 
the record button on a camcorder, the release of a 
pause button on a camcorder, or the beginning of a spe- 
cial effects sequence on the tape. An event-specific sig- 
nal may be initiated by the user to mark an event that 
the user wishes to identify during a recording sequence. 
The representative frame is identified by marking mod- 
ule 14 which is built into camcorder 12 and which is 
operable to place a coded instruction (reference mark 
64) on tape 34. Reference mark 64 is a pre-defined 
binary word. 

It should also be appreciated that the system and 
method of the invention may be incorporated into a VCR 
so that tapes recorded without the benefit of the instant 
invention may be marked so as to have a visual table of 
contents incorporated thereon. Such a VCR will include 
a marking module which is activated by a user of the 
system. Reference marks may be placed on a tape by 
the user, the tape is scanned and an index frame (IF) 
compiled and a video table of contents recorded at the 
beginning of the tape. Alternately, an IF is compiled in a 
MIC as the representative frames are identified by the 
user during the playback of a previously recorded tape. 

As previously noted, in some instances, a video 
cassette may contain a memory-in-cassette (MIC) 
mechanism 18, mechanism, such as a flash memory 
chip, which may be used to store the information asso- 
ciated with a representative frame. This portion of the 
description, however, will focus on the marking of repre- 
sentative frames on the tape in a video cassette not 
having a separate memory structure. 

After the video tape has been recorded, it may be 
scanned to determine the number and location of repre- 
sentative frames along the length of the video tape, 
block 66 in Fig. 3. This function is accomplished by 
scanning mechanism 22. which may be located in a 
camcorder, or in a docking station, which is provided for 
the camcorder and which has controls and a storage 
mechanism 24, in additional to those found in cam- 
corder 12. 

Features of the representative frames may be used 
in an interactive, content-based image/video database 
for search and retrieval of the video and/or its individual 
scenes, as well as in video editing. The representative 
frames are appropriately resized and tiled into a single 
index frame that may be recorded on the same video 
tape at a predetermined location for subsequent view- 
ing and/or output. 

As video tape 34 is scanned, representative frames 
and the location data therefor are read by scanning 
mechanism 22 and, in the preferred embodiment, are 
resized and arranged so as to allow the presentation of 
multiple representative frames in a single video image, 
. block 68. 

The resized and arranged representative frames 
are placed in storage mechanism 24 until all such repre- 
sentative frames are retrieved from the length of video 
tape 34. The resized and arranged representative 
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frames comprise an index frame, which is stored in an 
index frame memory (IFM) 70 within storage mecha- 
nism 24, and are subsequently recorded on video tape 
34 by a recording mechanism 13, which is a standard 
component of camcorder 12 or a VCR. Resizing and 5 
arranging may alternately be performed by a mecha- 
nism located in a camcorder. As previously noted, refer- 
ence mark 64 may be located in the maker's optional 
area of sub-code sector 50. The index frame may be 
compressed and recorded and stored in video sector w 
48, or in the maker's optional area of MIC 18, if it is sup- 
ported. 

DVCR standards require a "dead zone" on the first 
1 75mm of video tape following the tape leader. Absolute 
A/V track T 0 is located immediately after the 175mm 15 
length of tape. A feature of the instant invention is the 
reservation of the first 30 seconds of tape following track 
T 0 for use with the index frame of the invention The A/V 
tracks are sequentially numbered along the length of 
tape 34 from T 0 ...T n . 2 o 

As previously noted, peripheral storage 26 may be 
provided, and may take the form of a personal computer 
of some type. Peripheral storage 26, also referred to 
herein as peripheral index frame memory storage, may 
be used to edit and enhance the information contained 25 
in the representative frame and location data therefor, 
and may also be used to provide F/X. Peripheral stor- 
age 26 may also provide command and control func- 
tions for docking station 20, and may also provide the 
resizing and arrangement mechanism. 30 

Another feature of a camcorder incorporating the 
invention is the ability to turn the system off, if the user 
so desires, thereby directing the use of the first 30 sec- 
onds of tape to conventional recording. It should also be 
appreciated that the system and method of the inven- 35 
tion is adaptable to analog video systems, wherein ref- 
erence mark 64 may be incorporated onto a video or 
audio track for sensing by a scanning mechanism. The 
index frame may be generated by digital means and 
converted for recording onto an analog tape. 40 

Referring now to Fig. 4, a television monitor is 
depicted generally at 80. Monitor 80 includes a screen 
82 on which is displayed a plurality of representative 
frames as stored in an index frame memory and dis- 
played as a single multiple image video frame, generally 45 
at 84. Index frame memory display will generally have 
the same spatial resolution as a full DVCR frame, on the 
order of 720 X 486 resolution (525 line system). The 
index frame memory may contain one or more pages of 
representative frames, and may be located in cam- 50 
corder 12, in docking station 20, or in peripheral storage 
26. 

An enlarged representative frame is depicted gen- 
erally at 86 and would include an image portion 88 and 
a data portion 90. Data portion 90 may include location 55 
data relevant to image portion 88. A sound bite icon 92 
is provided which, assuming that all of the prerequisites 
have been met, will provide a sound clip which is asso- 



ciated with the representative frame and the several 
frames following the representative frame. The sound 
clip may be incorporated within a MIC structure, or 
placed in the VAUX, AAUX, or in the sub-code. 

In the event that peripheral IFM storage 26 is pro- 
vided, the user may edit the text displayed in data por- 
tion 90 of representative frame 86. 

As previously noted, some video cassettes compli- 
ant with the DVCR standard may contain an onboard 
memory structure, which is referred to as memory-in- 
cassette (MIC). The memory structure of a MIC is 
depicted generally at 100 in Fig. 5. The MIC structure is 
connected to camcorder 12 or to a VCR by a four-termi- 
nal connector using inter-integrated circuit (l 2 C) proto- 
col MIC structure 100 in the DVCR standard is divided 
into two main portions, the first being space or bank 0, 
depicted at 102, and the second being space 1, 
depicted at 104, having up to 255 banks. Each bank 
contains 64 Kbytes of memory space. Space 0 has a 
number of predetermined usages which are determined 
by the standards for consumer-use digital video. The 
remainder of space 0, and ail of space 1 , may be used 
for storage of data, sound, and video as defined by the 
manufacturer of the camcorder or VCR which uses a 
cassette equipped with MIC. The DVCR standard pro- 
vides for MIC of up to 16 Mbytes. It is expected that 
most commercially available cassettes will contain 4 or 
5 Mbyte MICs. 

In the case where a MIC is provided on the cas- 
sette, marking module 14 of camcorder 12 will still mark 
the tape with a reference mark, however, the associate 
representative frames immediately will be resized and 
arranged by resizing and arrangement mechanism 23, 
which in this embodiment, is located in camcorder 12, 
and placed in MIC 18, which also serves as the storage 
mechanism and index frame memory. Location informa- 
tion, in the form of the first track numbers of representa- 
tive frames, T f , is also recorded in the MIC. The index 
frame memory will immediately be available to the user 
and will be updated as new video and sound are 
recorded on the video tape, without the need to scan the 
entire video tape to compile the IFM. Playback of the 
index frame may be a continuous play of the first 30 sec- 
onds of tape, or, the index frame may be a single frame 
having multiple images with an instruction to cause the 
playback device to pause on that frame. It should be 
understood that there may be multiple index frames, 
each with multiple images therein, depending on the 
number of representative frames identified on the tape. 

Although a preferred embodiment, and one alter- 
nate thereof, have been described herein, it should be 
appreciated that variations and modifications may be 
made to this invention as defined in the appended 
claims. 

Claims 

1. A system (10) for determining representative 
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frames (56) of a video sequence captured on a 
video tape (34) comprising: 

a marking module (1 4) for marking and identify- 
ing the location, on a video tape (34), upon the 5 
occurence of a predetermined event, thereby to 
identify a representative frame (56) associated 
with the event on the recorded video tape (34) 
and to generate location data therefore; 
a storage mechanism (24) for storing plural 10 
representative frames and location data asso- 
ciated therewith; and 

a recording mechanism (13) for recording plu- 
ral representative frames. 

15 

2. The system (10) of claim 1 which further includes a 
scanning mechanism (22) for scanning said 
recorded video tape (34) having said representative 
frames (56) marked thereon to retrieve said repre- 
sentative frame and said location data therefrom. 20 

3. The system (10) of claim 1 wherein said marking 
module (14) is located in a camcorder (12) and said 
scanning mechanism (22), said storage mecha- 
nism (24) and said recording mechanism (13) are 25 
located in a docking station (20). 



11. The system (10) of claim 1 wherein said represent- 
ative frame and said location data are recorded on 
said memory unit in a maker's optional area. 

12. The system (10) of claim 1 wherein said marking 
module (14) is located in a camcorder (12) and said 
scanning mechanism (22), said storage mecha- 
nism (24) and said recording mechanism (13) are 
located in a docking station (20), and which further 
includes a peripheral storage mechanism (26). 

13. The system (10) of claim 1 wherein said predeter-. 
mined event is the beginning of a recording 
sequence. 

14. The system of claim 13 wherein said predeter- 
mined event is taken from the group of events con- 
sisting of pressing the record button, releasing the 
pause button, adding a special effect and initiating 
an event-specific signal. 

15. The system (10) of claim 1 which includes a re-siz- 
ing mechanism (23, 68) for re-sizing the represent- 
ative frames (56) so that plural representative 
frames (86) are displayed as a single multiple 
image video frame (84). 



4. The system (10) of claim 1 wherein said marking 
module (14) is located in a camcorder (12) and said 
storage mechanism (24) and said recording mech- 30 
anism (13) are located in a video tape cassette 
(16). 



5. The system (10) of claim 1 wherein said marking 
module (14), said scanning mechanism (22), said 35 
storage mechanism (24) and said recording mech- 
anism (72) are located in a camcorder (12). 

6. The system (10) of claim 1 wherein said marking 
module (1 4) is located in a camcorder (1 2) and said 40 
scanning mechanism (22), said storage mecha- 
nism (24) and said recording mechanism (13) are 
located in a video tape recorder. 

7. The system (10) of claim 1 wherein plural repre- 45 
sentative frames and said location data therefore 
are recorded on the video tape. 

8. The system (10) of claim 1 wherein a reference 
mark is recorded on said tape in VAUX (49) sector, so 

9. The system (10) of claim 1 wherein a reference 
mark is recorded on said tape in a sub-code sector 
(50). 

55 

10. The system (10) of claim 1 wherein said represent- 
ative frame and said location data are recorded in a 
memory unit in a video cassette (16). 



1 6. The system of claim 15 wherein said multiple image 
video frame (86) includes location data (90). 

17. A method for generating a multiple image repre- 
sentative frame comprising: 

marking the location of a representative frame 
(56) of video data on a video tape (34) on the 
occurence of a predetermined event; 
identifying location data associated with each 
representative frame (56); 
storing the representative frames and the loca- 
tion data associated therewith in an index 
frame memory; 

merging representative frames and the location 
data associated therewith into a single multiple 
image video frame; and 
recording the multiple image video frame. 

18. The method of claim 17 which further includes 
scanning a recorded video tape (34) for represent- 
ative frames and location data and wherein said 
recording step includes recording the multiple 
image video frame onto the video tape (34). 

19. The method of claim 18 wherein said recording 
includes recording the multiple image video frame 
into a memory unit (18) in a video cassette (16). 
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FIG.3 
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(54) System and method for determining representative frames of video captured by a video 
camera 



(57) A system and method of determining repre- 
sentative frames (56) of a video sequence to form a vis- 
ual table of contents for a video tape (34) is disclosed. A 
representative frame is identified on the occurence of 
an event, such as the beginning of a recording 
sequence, which may follow a record command, a 
pause-release command, some form of special effect 
command, or a user-initiated marking command. The 
representative frame, and location data associated 
therewith, may be marked along the length of a video 
tape (34) in a camcorder (12) or VCR, or, the represent- 
ative frame (56) may be identified and stored, along with 
the location data therefor, on a MIC (18). In the case 
where the representative frame location data is stored 
on the tape (34), a scanning mechanism (22) is pro- 
vided for scanning the recorded video tape to locate the 
representative frames (56). In either form of the inven- 
tion, multiple representative frames and their location 
data are placed in a storage mechanism (24), either 
temporarily or permanently, and then recorded at a sin- 
gle location, either at the beginning of the video tape 
(34) or in the MIC (18). A peripheral storage mechanism 
(28) may also be provided, in the form of a computer, 
which may be used to add additional information to the 
representative frame (56), and which may also be used 
in editing the video tape (34). 
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